Vision and neural control for an orange harvesting robot

We describe the system control architecture of a large, orange harvesting robot. This robot has two independent electrically driven telescopic arms mounted on a common platform which is itself held by a large hydraulic arm. This arm, in turn, is mounted on a tracked vehicle. The telescopic arms have cameras within the end-eeectors, that are used to detect and to measure the position and distance to fruit within the canopy of a tree. Most of the development and control software was implemented using the matrix-based Virtual Machine Language (VML))11]. This language was designed to implement neural networks, and has been extended and enhanced for robotic applications and the particular low level control requirements of the hardware. Three features of the VML language are central to this project. First, the language includes a wide range of matrix arithmetic operators which facilitate the implementation of image processing, coordinate transformation, neural networks, and data transfers. Second, the portability of the control language, has permitted design and development to take place on both a Unix workstations and under the realtime operating system (OS9) used on the robot hardware. Third, all of the data transfer and hardware control operations share a common interface, based on a small set of functions. These device drivers provide the interface to frame grabbers, motor drivers, digital interface electronics, proximity detectors, and le handling. Exactly the same interface is used to implement interprocess communications with display and monitoring tools. In spite of the widespread economic importance of automatic harvesting, and many concurrent research projects (e.g. 7] 8] 5]), no commercial orange harvesting system is available. There are many problems that a robot-based harvester must overcome. Oranges must be individually picked and they vary signiicantly in shape, size and colour. An economically successful robotic orange harvesting system must detect and harvest oranges at almost the accuracy achieved by humans and must be faster. Oranges are distributed in a strip about one meter deep within the canopy, in a completely unstructured environment. Detecting and harvesting the fruit involves many complex operations in image analysis, motion planning and the control of arm movements. Automatic visual identiication of oranges is further complicated by variation in lighting conditions from bright sunlight on the outer parts of the canopy to deep shadow within the canopy. The oranges often have a dark background, but can also have an illuminated background. Also oranges often grow in clusters, and …